CoMER: Modeling Coverage for Transformer-Based Handwritten Mathematical Expression Recognition

The Transformer-based encoder-decoder architecture has recently made significant advances in recognizing handwritten mathematical expressions. However, the transformer model still suffers from lack of coverage problem, making its expression recognition rate (ExpRate) inferior to RNN counterpart. Coverage information, which records alignment information past steps, proven effective models. In this paper, we propose CoMER, a that adopts decoder. Specifically, novel Attention Refinement Module (ARM) refine attention weights with without hurting parallelism. Furthermore, take extreme by proposing self-coverage and cross-coverage, utilize current previous layers. Experiments show CoMER improves ExpRate 0.61%/2.09%/1.59% compared state-of-the-art model, reaches 59.33%/59.81%/62.97% on CROHME 2014/2016/2019 test sets. (Source code is available at https://github.com/Green-Wood/CoMER )